ANALYTIC HYPOELLIPTICITY FOR A CLASS OF 
SUMS OF SQUARES OF VECTOR FIELDS WITH 
NON-SYMPLECTIC CHARACTERISTIC VARIETY 



ANTONIO BOVE, MAKHLOUF DERRIDJ, AND DAVID S. TARTAKOFF 
Abstract. Recently, N. Hanges proved that the operator 



and yet fails to be analytic hypoelliptic 'in the strong sense' in 
any neighborhood of the origin (there is no neighborhood U of the 
origin such that for every open subset V of U and distribution 
u in U, Pu analytic in V implies that u is analytic in V). Here 
dt = d/dt, c?eo) = ^e{x) — x\d/dx2 — x%d/dx\,x — {x\,X2), and 
= d 2 /dxf + d 2 jdx\. We give a very short L 2 proof of this result, 
obtained jointly with A. Bove and M. Derridj, which generalizes 
easily and suggestively to other operators. It is striking that these 
operators have non-symplectic characteristic varieties. Finally we 
point out that these results are consistent with Treves' conjecture. 



1. Introduction and Generalizations 
In is recent paper Hanges considered the operator 



in IR 3 where dg( x ) = Xid/dx 2 — x 2 d/dxi and made the interesting dis- 
tinction between analytic hypoellipticity in the germ sense and a.h.e. 
in the strict sense. Hanges proved that the operator P was not analytic 
hypoelliptic in the strict sense in any open set U containing the origin, 
i.e., did not have the property that for any open subset V of U, if Pu is 
analytic in V then so is the solution u, yet had the property that if Pu 
was analytic in some neighborhood of the origin then the so was u in 
a (possibly smaller) neighborhood of the origin. He ties this result to 
the conjecture of Treves concerning the Poisson strata of the operator 
P, namely that if one writes P = ^2 1 Xj, and considers the successive 

strata where 1) all Xj vanish, 2) all Xj and their first brackets vanish, 

l 
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3) all Xj and their first and second brackets vanish, etc., then the op- 
erator should be analytic hypoelliptic in the strict sense if and only if 
all these strata are symplectic. In the case of the particular operator 
being considered here, not even the characteristic variety is symplectic, 
being given by t = r = x\^ 2 — ^2^1 = 0. 

Here we give a very elementary, and flexible, proof of the affirmative 
part of his result and argue that the negative part is entirely reasonable 
as well, though we avoid entirely the mention of so-called Treves curves, 
which foliate the characteristic variety of P, in our proof. 

The generalizations we consider may be motivated by observing that 
while the "added" term in P, which suggests the celebrated non- 
analytic hypoelliptic example of Baouendi and Goulaouic, 

(1-2) PBO = % + 1*% + Si = j2%, 

1 

differs from this example in one essential factor — the integral curves of 
d y are non-compact yet those of dg which start close to the origin remain 
close. Hence a propagation of singularities result may be rephrased in 
terms of a germ result on analyticity. 

To put the matter differently, the L 2 proof of propagation of singu- 
larities for Pbg hinges (writing iD = d) on the fact that in estimating 
localized high derivatives of a solution u in the x-direction, ip(x, y)D^u, 
via the L 2 a priori estimate (we take f independent of t since for t 7^ 0, 
the operator is elliptic), one encounters and cannot avoid the derivation 
(and bracket) 

J2 \\Z^Dluf L , < \{P BG ipDlu,ipD p x u) L .\ 
j 

S£ll[^,^]«||£ a + ... 

< \\<f'D p x u\\h + ■■■ 
When the value of j is 3, i.e, we are trying to estimate D x derivatives 
of u and encounter a y-derivative of ip with no gain in the number of 
rc-derivatives, we cannot proceed, even with Ehrenpreis type localizing 
functions, to obtain analytic growth. Unless, of course, the y-derivative 
of the localizing function is supported in a region where the solution is 
known to be analytic already. 
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However, if the localizing function <p could be written as a function 
independent of the y-variable as well, this situation would not arise 
and analyticity would follow (after some calculation, admittedly, but 
elementary calculations with no sophisticated ingredients.) 

This is what occurs when the open set under consideration is global 
in the "y-direction" , as in proofs of analyticity which are local in some 
variables and global in others, as on a tube or torus, or when the vector 
field D y is replaced by a vector field whose integral curves remain in any 
neighborhood of the point under consideration, as in Hanges' example, 
where D y is replaced by Dq. 

Thus the following generalization of Hanges' example suggest them- 
selves rapidly: in (t,x) G M. e x M fc , and with dj = d/dxj, 

k 

(1.3) Pi = A t + \t\ 2 A x + aij(x,t){xidj - x^f 

for positive definite and analytic matrix valued function a^. Note the 
critical feature of this operator that the Laplacian in x commutes with 
each of the angular operators Xidj —Xjdi. Actually, in terms of estimates 
what is crucial is that there be a C u basis, {Xj}, of vector fields in the 
x variables in in ¥L k \ {0} such that any bracket, [Xj,Xide — X(dj\ be a 
linear combination of the angular vector fields x^j — Xjdi (over which 
we have coercive control). 

We also remark that when a^x, t) = 1, then it is not hard to see that 
the last sum is a constant multiple of the Laplace Beltrami operator on 
the unit sphere. 

Still more generally, let us consider k vector fields Xi, . . . ,Xk in 
the x— variables with analytic coefficients (of x, t) and s vector fields 
Y]_, . . . , Y s in the x— variables which may be singular but have analytic 
coefficients . Let xq G M fe be a fixed point and denote by U C M fe an 
open neighborhood of Xq. Without loss of generality we may suppose 
that xq = 0. We assume that 

1 - The {d/dt m , Xj} j=i,...,k span the tangent space on every point 

m=i,..'.,e 

(t, x) G R e x U, with x ^ 0. 

2 - Y\ , . . . , Y s have a compact closed family of integral manifolds 

which foliate U . 
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3 - We assume that the following commutation relations hold: 

(1.4) [j—, X e ] = lin. comb, of the Y, — , and tX, Vm, £, 

and for some positive definite quadratic form A in the X^s, 
with coefficients independent of the t variables, 

d 

(1.5) [A, Ye] = quadratic expr. in the Y, — , and tX, W. 
Consider then the operator 

k s 

(1.6) P 2 = A t + \t\ 2 (HjXiXj + h ^ YiY ^ 

i,j=l i,j=l 

where dij(t,x) and bij(t,x) are positive definite matrices. We will 
show that we may argue as in the particular case to obtain the result 
that P 2 is analytic hypoelliptic in the sense of germs at the origin. 

Note that assumption 2 implies that we may choose a localizing func- 
tion constant on the integral curves of Y\, . . . Y s , of Ehrenpreis type, 
identically equal to one on any given compact subset of U but vanish- 
ing outside of U. 

We state our theorem: 

Theorem 1.1. Let us consider the operator P 2 as in M.b]) . where the 
coefficients real analytic in a neighborhood of the origin. Let 

U be a neighborhood of the origin with the properties in Assumptions 
1-3 above. Let P 2 u = f hold on the same open set U, with f e 0^(17). 
Then u is also in 0^(11). 

2. Proof in the case of Hanges' operator (11.11) 

As remarked above, we may take localizing functions to be indepen- 
dent of t, since were a derivative in t to land on such a localizer, one 
would be in the region where the operator was clearly elliptic and the 
analyticity of the solution u is well known. We denote such an Ehren- 
preis type localizing function by (p(x) = <Pn(x) subject to the usual 
growth of its derivatives: l-D^V 9 ! — C' Q ' +1 A^' a ' for |a| < N, where the 
constant C is (universally) inversely proportional to the width of the 
band separating the regions where (p = and if = 1. 

Next, since P is C°° hypoelliptic we may assume that u is smooth 
and proceed to obtain estimates for D\u and D^u near 0. 
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The a priori estimate for P, while subelliptic, is more importantly 
maximal: for v G C^°, 

2 

(2.1) \\D t v\\l, + \\tD Xi v\\l> + \\De {x) v\\h (+ IMIi/ 2 ) 
i 

<C\(Pv,v)\ + C\\v\\ 2 L ,. 

Setting v = <p>D P u, to begin with, we obtain 

2 

\\D t pD P u\\h + E \\tD Xj ^u\\h + \\D e(x) pD P u\\h (+ yDMl/2) 
1 

<C\(PpD P u,pD P u)\ + C\\pD P u\\ 2 L2 

4 

1 

Now crucial in the brackets are the quantities (recall that we may take 
ip independent of t, and clearly to localize in x we may take it to be 
purely radial in (xi,x 2 ) i.e. we choose p to be constant on the integral 
curves of X 4 ), so that X A p = 0, 

[X 1 ,pD^] = [X 4 ,pD^]=0, 

and 

[Xj, pD p ] = tp'D P - ppD x D p ~\ J = 2, 3. 

In the first case, we may ignore the factor t and recognize the passage 
from one power of D t to a derivative on <p as an acceptable swing, 
which, upon iteration, will lead to C P+1 N P ~ C p+1 p\ when p ~ N. The 
second term takes two powers of D t (e.g., X\ from the estimate and 
one power of D t and produces a factor of p and a 'bad' vector field D x . 
Iterating this will yield p\\D^/ 2 u ~ p^}l 2 D v J 2 u on the support of </?. 
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On the other hand, setting v = ipD x .u, with perhaps q — p/2, or, 
better, v = (pA%/ 2 u, where we write A x = £\ D 2 ., 

2 

|| D t <pA*/ 2 u\\ 2 L2 + \\tD Xj <pAfu\\l 2 + || De ix) <pA*' 2 u\\* 
1 

< C\{P<pAl/\<pAfu)\ + cyAfu\\h 

4 

i 

+ C\\<pAfu\\ 2 L2 , 

and now the crucial brackets are 

[X^Af] = 0,[X 4 2 ,y,Af] = 

and 

[X 2 , <pAf] = 2X jt <p'Af - t 2 ^Af, j = 2,3 

(where we have used rather heavily the fact that X^ip = since ip 
depends only on x, and radially so, and that in fact [Dg, A x ] = 0.) 
This last line leads to two kinds of terms, namely, for j = 2, 3, 

(2X,V 'Afu, tpAfu) 

and 

Morally, these terms show the correct gain to lead to analytic growth 
of derivatives, namely one must think of tA 1 / 2 as an Xj with j = 2 
or j = 3, and so in the first term above one merely integrates by part 
noting that X* = —Xj and obtains, after a weighted Schwarz inequality, 
a small multiple of the left hand side of the a priori inequality and the 
square of a term with one derivative on ip and tA 1 / 2 and q reduced 
by one, though one more commutator is required to make the order 
correct, and this will introduce another derivative on (p and q again 
decreased by one unit, etc. The second term is of a different character, 
though the same observation reduces us essentially to 

{Xp^A^- l),2 u,X^- l),2 u) 
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in which instead of each copy of ip receiving one derivative, we have 
two derivatives on one copy and none on the other. Fortunately, the 
Ehrenpreis-type cut-off functions may be differentiated not merely iV 
times with the usual growth but 2N or 3iV with no change — so in the 
above inner product we include a factor CN with the copy of ip which 
remains undifferentiated and a factor of (CN)' 1 with the other. The 
estimates work out just as before. 

3. Proof in the general case (jl.6|) 

The general case is not more complicated than the first, simplest 
case, with A t = J2m=i ^t m requiring us to consider each t-variable 
separately; A x is replaced by Ylij=i a ij(A x )XiXj and the square of the 
angular derivative by the sum Ylij=i hj(t, x)YjYj. Thus we merely give 
a brief sketch of the proof. 

We have an a priori estimate of the form 

e ik s 

(3.i) £pvii 2 +£E ii v^n 2 + £ ii^n 2 + ihi 2 /2 

j=l j=l h=l 3=1 

<C(\(P 2 u,u)\ + \\u\\ 2 ). 

Let us write, as introduced above, 

A = a positive definite quadratic expression in the Xi 

and 

k 

A = } j aij(t,x)XiXj. 

i,i=l 

Analogously to what has been done before we denote by ip a cut-off 
function of Ehrenpreis type, constant on the integral manifold of the 
fields Yi, . . . , Y s and independent of t. 

The problem of estimating the growth rate of the derivatives of u 
then reduces to estimating 

for every natural number q < N, where \d a ip\ < C 1+ ^N^, for < 
H < 3A^. 

We have thus to examine the structure of the commutator 
[P 2 , ipA g/2 } = [At + \t\ 2 A + B, ipA q/2 ], 
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where we wrote 

s 

B= J2b ij {t,x)Y i Y r 
The above quantity becomes: 

[A f + \t\ 2 A + B, ipA q / 2 ] = |<£>[A t) A]A q/2 ~ 1 + \t\ 2 [A, v ]A^ 2 

+ | |t| 2 <p[A, A]A g/2_1 + <p[B, A q/2 ] =T 1 +T 2 + T 3 + T 4 , 

modulo lower order terms whose treatment is easier. Let us look at each 
term in the above formula, denoting by 'elliptic' any term which con- 
tains, in the inner product, two factors of the form maximally estimated 
by the operator, namely two factors each of the form Y,tX, or d/dt. 
Such terms will be subject to the a priori inequality (after an integration 
by parts) in a recursive manner and will cause little trouble. 

3.1. T\. Since the commutator appears in a scalar product, taking one 
t-derivative to the other side, we have to estimate, for some coefficient 
a(x), 

<?l MA. , A] A 9/2 ~V D ts <pA q/2 u) \~q\{(p [A, , aX}A {q ~ 1)/2 u, D ts tpA q/2 u) | . 

But by (jl.4j) . this bracket is elliptic, hence the factor of q balances the 
decrease in the exponent of A and will iterate analytically. 

3.2. T 2 . We have to estimate 

\(\t\ 2 [A, v ]A q / 2 u, V A^ 2 u}\. 

The commutator in the left hand side factor of the above scalar product 
leads to an expression of the type 

2|(|t| 2 X^'A 9/2 M ,^A 9/2 M)| ~ 2\(Zip'A {q - l)/2 u,ZifA q/2 u)\ 

with elliptic Z modulo (easier) lower order terms. Here we used just 
the form of A. A weighted Schwarz inequality shows that this term it- 
erates analytically, since with the Ehrenpreis-type localizing functions, 
a derivative on if balances a decrease in q. 
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3.3. T 3 . We have to estimate the scalar product 




Since 

k k 



(3.2) [A, A] = ]T KXiXi,a,X a X p \ 



i,j=l a,/3=l 



$>X 2 {X or ^} 



again modulo lower order terms, using assumption 3, part 1. 

Now one of the X factors raises q/2 — 1 to q/2 — 1/2 = (q — l)/2, 
with the factor of q balancing the decrease from q to q — 1, and the 
other two X's (or one X and one which is better, combine with t 2 to 
produce Z 2 , with Z elliptic. Thus this inner product, as well, iterates 
analytically. 



the estimate of T 4 boils down to computing the commutator [B, A] and 
estimating the resulting terms. But this goes as before in view of the 
second part of assumption 3, since the bracket contains a product of 
three vector fields, two of which are elliptic and one serves to convert 



This ends the proof of the Theorem in the general case. 

4. Remarks vis-a-vis the conjecture of Treves 

The conjecture of Treves states, in this context, that the operator 
P should be analytic hypoelliptic at the origin if and only if all layers 
of the Poisson stratification are symplectic. The first of these layers is 
the characteristic manifold, which is patently non-symplectic in all of 
these cases. And the operators are analytic hypoelliptic in the sense of 
germs at the origin. 

The distinction is crucial. For to contradict Treves' conjecture, there 
would have to exist an open neighborhood V of the origin, in which one 
could have analytic data with a non-analytic solution. And that may 
still well be the case. What we have shown is that for neighborhoods of 



3.4. T 4 . Since 



A ,/2-l tQ A ( 9 -l)/2_ 
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the origin of a certain geometry relative to the operator, analytic data 
forces analyticity of the solution. 

It is well known however that in the case of the operator P2 there 
is in general propagation of the analytic wave front set (or rather of 
the analytic regularity) along the Hamilton leaves of the characteristic 
manifold (which are non trivial in this case), see e.g. pp. 

From the above proof we may see that the analyticity of the solution 
is forced, in some "adapted" open set, by a "global" phenomenon, that 
might be described by saying that the analytic singularities of the so- 
lution in the open set under consideration comes from points outside 
the open set lying on some Hamilton leaf of the characteristic manifold. 
This is actually prevented by the "large scale" geometry of the open 
set. 

It can thus be asserted that, far from being in contradiction with 
Treves' conjecture, the present result is in complete agreement with it 
and points out the importance of the geometry of the non-symplectic 
strata of the Poisson- Treves stratification. 
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